Analyzing Facial Expressions for Virtual

ideo coding applications such as video conferencing, telepresence, and tele-teaching have attracted considerable interest in recent years. Transmitting image sequences typically imposes high demands on storage, processing power, and network bandwidth. Common networks like POTS (Plain Old Telephone System), ISDN (Integrated Services Digital Network), and many computer networks provide only low bit rates and cannot handle transmitting uncompressed video. Therefore, encoding the sequences proves essential. Highly efficient coding schemes like H.263 and MPEG-1 and 2 achieve compression factors of about 1:100 with acceptable quality. Using model-based coding techniques further improves compression while simultaneously providing us with the possibility of manipulating a scene's content interactively. Model-based coding aims to create a virtual 3D world with distinct 3D objects described by their shape and texture. In this world we only have to specify how the objects move and deform. The objects' shape transmits only once, and we can then describe the scene in all the following frames with few parameters specifying the objects' position and deformation. We reconstruct the 2D image sequence by rendering the virtual world using computer graphics techniques. Model-based coding of image sequences requires 3D models for all objects in the scene. Therefore, we must put some restrictions on the scene content. In this article we focus on head and shoulder scenes for video telephone and conferencing applications. Let's see how this process works. A speaking person sits in front of the system while a single camera records video frames. The encoder analyzes the incoming frames and estimates the person's 3D motion and facial expressions. We obtain a set of facial-expression parameters that describe (together with the 3D model) the person's current appearance. We only need to encode and transmit a few parameters, leading to very low data rates (typically less than 1 Kbps). At the decoder we use the parameters to deform our head model according to the person's facial expressions and approximate the original camera frame by rendering our 3D model. Figure 1 shows the model-based coder's principal structure. We can easily manipulate and interact with the scene content once segmented into single 3D objects. For example, the eye contact problem that occurs in conventional video telephone systems resolves if we position the virtual camera inside the monitor that displays the image of the person with whom we're talking. Or, we can look around objects if we move the synthetic camera according to the eye trajectory the …